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October  18, 1999 

The  Honorable  Jerry  Lewis 
Chairman,  Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 

Dear  Mr.  Chairman: 

Complete  and  thorough  Year  2000  (Y2K)  testing  is  essential  to  provide 
reasonable  assurance  that  new  or  modified  systems  process  dates  correctly 
and  will  not  jeopardize  an  organization’s  ability  to  perform  core  business 
operations  after  the  millennium.  This  is  especially  true  for  the  Department 
of  Defense  (DOD)  which  relies  on  a  complex  and  broad  array  of 
interconnected  computer  systems — including  weapons,  command  and 
control,  satellite,  inventory  management,  transportation  management, 
health,  financial,  personnel  and  payment  systems — to  carry  out  its  core 
business  functions  and  military  operations. 

At  your  request,  we  initiated  a  review  of  the  effectiveness  of  DOD’s  efforts 
to  perform  Year  2000-related  end-to-end  tests  for  its  major  business 
functions,  including  Health  Affairs,  Communications,  Personnel,  and 
Logistics.  Together,  these  functional  areas  are  performing  thousands  of 
end-to-end  tests  to  ensure  that  key  business  processes  and  systems  can 
continue  operating  into  the  year  2000.  Specifically,  for  each  functional  area, 
we  analyzed  reported  information  on  the  status  and  progress  of  all  test 
events.  We  also  selected  and  reviewed  a  critical  test  event  in  each 
functional  area  to  determine  whether  it  was  planned  and  managed  in 
accordance  with  our  Year  2000  testing  guide.1  On  September  14, 1999,  we 
briefed  you  on  the  results  of  our  review.  This  report  provides  a  summaiy  of 
our  briefing  and  a  recommendation  to  Defense  for  strengthening  oversight 
of  end-to-end  testing  for  the  Communications  functional  area.  Subsequent 
to  our  briefing,  logistics  officials  submitted  additional  information  on  the 
inclusion  of  installation  telecommunications  providers  in  related  test 
events.  We  have  clarified  the  briefing  slides  to  reflect  this.  These 
clarifications,  however,  do  not  affect  our  overall  conclusions  and 
recommendation.  The  briefing  slides  are  presented  in  appendix  I,  and  our 


1  Year  2000  Computing  Crisis:  A  Testing  Guide 


,  November  1998). 
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objectives,  scope,  and  methodology  are  in  appendix  II.  The  Office  of  the 
Assistant  Secretary  of  Defense  provided  written  comments  on  a  draft  of 
this  report.  These  comments  are  discussed  at  the  end  of  this  report  and 
reprinted  in  appendix  III.  We  performed  our  audit  work  from  March 
through  September  1999  in  accordance  with  generally  accepted 
government  auditing  standards. 


Results  in  Brief 


Because  Year  2000  conversions  often  involve  numerous  large 
interconnecting  systems  with  many  external  interfaces  and  extensive 
supporting  technology  infrastructures,  Year  2000  testing  should  be 
approached  in  a  structured  and  disciplined  fashion.  Our  Year  2000  guidance 
recommends  that  in  planning  and  managing  end-to-end  tests,  agencies 
define  test  boundaries,  secure  the  commitment  of  data  exchange  partners, 
prepare  test  procedures  and  data,  define  exit  criteria,2  and  document  test 
results,  among  other  steps.  Each  of  the  individual  test  events  we  attended 
and  reviewed  within  the  four  functional  areas  generally  satisfied  the  key 
processes  that  our  Year  2000  test  guide  defines  as  necessary  to  effectively 
plan,  conduct,  and  report  on  end-to-end  testing.3  Moreover,  while  the 
events’  respective  approaches  to  implementing  the  key  processes  varied, 
these  differences  were  appropriately  based  on  consideration  of  the  event’s 
scope  and  complexity. 

In  addition,  overall  end-to-end  test  efforts  within  three  of  the  four 
functional  areas  were  reported  to  be  largely  on  schedule  and  expected  to 
be  completed  by  October  1999.  However,  at  the  time  we  briefed  the 
Communications  functional  area  on  the  results  of  our  review,  it  could  not 
provide  complete  progress  information.  While  information  was 
subsequently  provided  by  Communications,  it  showed  that  the  functional 
area  had  not  yet  developed  plans  to  test  31  mission-critical  systems.  We  are 
making  a  recommendation  to  Defense  to  ensure  that  these  systems  are 
tested  or  that  there  is  adequate  justification  for  their  exclusion  from  end-to- 
end  test  events.  While  Defense  only  partially  concurred  with  this 
recommendation,  it  provided  information  showing  the  status  of  the 
systems  in  question.  We  did  not  verify  this  information. 


2Exit  criteria  are  test  conditions  or  requirements  for  successfully  completing  testing. 

3Our  observations  are  limited  to  the  specific  events  we  witnessed,  and  we  cannot  draw 
conclusions  regarding  end-to-end  testing  from  an  overall  functional  area  perspective. 
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Background 


In  August  1998,  the  Deputy  Secretary  of  Defense  recognized  the  need  to 
ensure  that  various  key  lines  of  business  or  functional  areas  within  the 
department  could  continue  to  operate  effectively  at  and  after  the  turn  of 
the  century.  Therefore,  the  Deputy  Secretary  directed  Office  of  the 
Secretary  of  Defense  focal  points,  known  as  Principal  Staff  Assistants 
(PSAs) ,  to  verify  that  all  functions  would  be  unaffected  by  Year  2000  issues. 
In  doing  so,  the  PSAs  were  to  (1)  document  mission-critical  functions  and 
systems  supporting  those  functions,  (2)  coordinate,  facilitate,  and  monitor 
Year  2000  end-to-end  test  and  evaluation  activities  of  services,  agencies, 
and  commands,  and  (3)  in  some  cases,  conduct  Y2K  end-to-end  functional 
testing. 

The  purpose  of  end-to-end  testing  is  to  verify  that  a  defined  set  of 
interrelated  systems,  which  collectively  support  an  organizational  core 
business  area  or  function,  interoperate  as  intended  in  an  operational 
environment  (either  actual4  or  simulated) .  These  interrelated  systems 
include  not  only  those  owned  and  managed  by  the  organization,  but  also 
the  external  systems  with  which  they  interface  or  that  otherwise  support 
the  core  business  area  or  function. 

The  boundaries  for  end-to-end  tests  can  vary  depending  on  a  given  business 
function’s  system  dependencies  and  criticality  to  the  organizational 
mission.  Therefore,  in  managing  end-to-end  test  activities,  it  is  important  to 
analyze  the  interrelationships  among  core  business  functions  and  their 
supporting  systems,  and  the  mission  impact  and  risk  of  date-induced 
systems  failures  and  to  use  these  analyses  to  define  test  boundaries.  It  is 
also  important  to  work  early  and  continually  with  functional  partners  to 
ensure  that  related  end-to-end  test  activities  are  effectively  coordinated 
and  integrated.  As  highlighted  in  table  1,  our  Year  2000  test  guide,  which 
has  been  adopted  by  the  Office  of  Management  and  Budget  (OMB), 
recommends  that  federal  agencies  take  the  following  actions  in  planning 
and  managing  end-to-end  tests. 


( Risks  of  testing  in  the  production  environment  must  be  thoroughly  analyzed  and 
precautions  taken  to  preclude  damage  to  systems  and  data. 
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Table  1:  Summary  of  Recommended  End-to-End  Test  Management  Processes 

Define  the  system  boundaries  of  the  end-to-end 
test(s) 

Agencies  should  define  boundaries  for  the  end-to-end  test  based  on  an  assessment 
of  their  mission-critical  business  functions,  inter-  and  intraorganization  system 
dependencies,  as  well  as  the  probabilities  and  impacts  of  any  of  these  systems 
suffering  a  date-related  failure. 

Secure  the  commitment  of  data  exchange 
partners 

Because  end-to-end  testing  addresses  business  areas  or  functions  that  involve 
multiple  internal  and  external  organizations,  participation  by  all  key  data  exchange 
partners  should  be  solicited  and  obtained. 

Establish  an  interorganizational  test  team 

A  team  composed  of  representatives  from  each  of  the  organizations  participating  in 
the  test  should  be  formed  to  manage  the  planning,  execution,  and  reporting  of  the 
test. 

Confirm  Year  2000  compliance  of 
telecommunications  infrastructure 

In  order  to  execute  end-to-end  testing  and  ensure  that  all  systems  in  the  chain  of 
support  to  core  business  areas  function  as  intended,  agencies  should  ensure  that  the 
telecommunications  infrastructure  that  interconnects  the  systems  is  compliant  and 
ready  for  testing. 

Schedule  and  plan  end-to-end  test(s) 

A  plan  should  be  developed  specifying  key  tasks  and  requirements  for  test  planning, 
execution,  and  validation  as  well  25  milestones  and  resources  associated  with 
performing  these  tasks. 

Prepare  end-to-end  procedures  and  data 

Interorganizational  test  procedures  and  data,  including  steps,  cases,  and  input 
conditions  that  verify  the  correct  handling  of  critical  dates,  should  be  prepared  and 
approved  by  team  representatives. 

Define  end-to-end  test  exit  criteria 

The  conditions  or  requirements  for  successfully  completing  end-to-end  testing  need  to 
be  established. 

Execute  end-to-end  test(s) 

Tests  should  be  executed  in  accordance  with  established  plans  and  procedures. 

Document  test  results 

Test  results  should  be  documented  so  that  the  data  can  be  used  to  validate  that  test 
exit  criteria  had  been  met  and  to  assess  and  correct  problems  discovered  during  the 
testing. 

Correct  Year  2000  defects 

On  the  basis  of  interoganization  specified  criteria,  such  as  defect  severity  and  test 
exist  criteria,  defects  identified  during  the  test  should  be  prioritized  and  corrected. 

Ensure  that  end-to-end  test  exit  criteria  are  met 

Test  results  should  be  compared  to  test  exit  criteria  to  ensure  that  specified  conditions 
are  met. 
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The  table  below  explains  how  the  four  functional  areas  included  in  our 
review  approached  their  end-to-end  tests. 

Table  2:  Summary  of  the  Four  Functional  Area  Test  Approaches 

Function 

Description  of  decomposition 

Health  Affairs 

Health  Affairs  divided  its  function  into  three  business  processes:  patient  care,  patient  administration,  and 
medical  logistics.  Health  Affairs  then  broke  down  each  process  into  several  sub-processes,  termed  “threads.” 

Communications 

Because  Communications  cross  cuts  all  functional/operational  areas,  Communications  is  testing  based  on 
system  user.  Therefore,  Communications  divided  its  function  into  263  mission-critical  systems.  Various  military 
services,  Defense  agencies,  and  commanders-in-chief  own  these  systems. 

Personnel 

Personnel  divided  its  function  into  six  areas:  Army,  Navy,  Air  Force,  Marine  Corps,  Civilian,  and  DEERS/RAPIDS 
(personnel  systems).  Personnel  then  broke  down  each  area  into  sub-processes,  termed  “threads.”  Personnel  is 
not  conducting  its  own  end-to-end  tests.  Instead,  Personnel  is  participating  in  and  observing  service-level 
testing. 

Logistics 

Logistics  divided  its  function  into  four  business  processes:  requisition,  receipt,  shipment,  and  inventory  control 
and  asset  status.  Logistics  then  broke  down  each  process  into  several  sub-processes,  termed  “threads.” 

Logistics  tested  these  four  processes  in  two  phases:  intracomponent  (within  each  military  service  or  Defense 
agency)  and  intercomponent  (joint  testing  with  military  services  and  Defense  agencies). 

The  test  events  we  selected  from  each  area  to  review  ranged  from  a  simple 
test  involving  two  information  systems  located  within  one  organization  to 
an  intricate  test  of  DOD’s  voice  and  data  telecommunications  networks 
involving  several  commands  and  multiple  systems.  Specifically,  the  Health 
Affairs  test  event  we  reviewed  assessed  the  ability  of  two  interfacing 
systems  to  issue  and  process  blood  requests  after  the  calendar  year 
rollover.  The  Communications  test  event  that  we  observed  was  a  portion  of 
a  larger  test  and  assessed  whether  voice  communications  could  be  sent 
from  Fort  Monmouth,  New  Jersey,  to  St.  Louis,  Missouri,  using  DOD 
telecommunications  networks  and  equipment  and  whether  messages  could 
be  exchanged  using  the  Defense  Messaging  System  from  the  Strategic 
Command  to  the  Atlantic  Command.  The  Personnel  test  event  assessed  the 
Army’s  ability  to  create  active  duty  units  for  deployment  from  the  Army 
Reserve  and  Army  National  Guard.  Finally,  the  Logistics  event  focused  on 
intercomponent  testing — between  the  Army,  Air  Force,  Navy,  Marine 
Corps,  and  Defense  Logistics  Agency — and  was  designed  to  verify  the  Year 
2000  readiness  of  17  of  the  53  total  logistics  requisition  and  receipt 
processes. 
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End-to-End  Tests 
Reported  to  Be  on 
Schedule 


Available  information  for  the  respective  areas  indicates  that,  as  of  August 

1999,  end-to-end  tests  were  largely  on  schedule  and  expected  to  be 

completed  by  October  1999.  In  particular, 

•  Health  Affairs,  which  had  three  primary  business  processes,  completed 
testing  for  two — patient  care  and  patient  administration  business 
processes — and  was  on  schedule  to  complete  tests  for  the  third — 
medical  logistics — by  the  end  of  September  1999. 

•  Personnel  tests  for  the  Army,  Air  Force,  and  Civilian  areas  had  been 
completed,  while  the  Navy  tests  were  scheduled  to  be  done  October  17, 
1999.  The  Marine  Corps  was  behind  schedule  on  one  test.  However,  it 
completed  the  test  by  September  9,  1999. 

•  Logistics  intra-  and  intercomponent  tests,  which  involve  four  primary 
business  processes — requisition,  shipment,  receipt,  and  inventory 
control  and  asset  status — had  been  completed  for  intercomponent 
transactions.  Tests  were  scheduled  to  be  done  by  the  end  of  August  1999 
and,  according  to  Logistics  officials,  were  completed  on  schedule. 

•  When  we  briefed  the  Communications  functional  area  on  the  results  of 
our  review  in  July  1999,  it  was  unable  to  provide  progress  information 
on  all  of  its  263  mission-critical  systems.  Subsequently,  Communications 
reported  that  77  mission-critical  systems  had  completed  testing  and  155 
systems  did  not  require  testing.5  The  functional  area  also  reported  that 
the  remaining  31  mission-critical  systems6  did  not  yet  have  plans  for 
testing  and  were  considered  to  be  behind  schedule. 


Selected  End-to-End 
Test  Events  Were 
Managed  According  to 
GAO  Guidance 


We  selected  one  test  event  from  each  functional  area,  determined  whether 
the  key  processes  outlined  in  our  Year  2000  testing  guide  were  followed 
and  found  that  DOD  had  completed  the  majority  of  the  processes  called  for 
in  the  guide.  For  example,  for  the  four  test  events  reviewed,  DOD  had 
defined  test  boundaries,  defined  exit  criteria  that  would  be  used  to 
determine  when  a  test  was  successfully  completed,  and  described  how  the 
test  results  would  be  documented.  While  the  event  s  respective  approaches 
to  implementing  the  key  processes  varied,  these  differences  were  based  on 


5The  Communications  function  considers  systems  that  do  not  require  end-to-end  testing  to 
be  developmental  systems,  those  that  do  not  process  dates,  and  stand-alone  systems. 

According  to  Communications  officials,  some  of  these  systems  are  satellite  and  control 
systems,  which  may  require  waivers. 


Page  8 


GAO/AIMD-OO- 1 2  DOD  Y2K  End-to-End  Testing 


B-283564 


the  consideration  of  the  event’s  scope  and  complexity  and  inherent 
business  risk.  Our  test  guidance  permits  such  differences  when  justified  on 
the  basis  of  business  value  and  risk. 

Table  3  summarizes  the  results  of  our  review.  As  the  table  notes,  of  the 
possible  44  key  processes  spanning  the  4  test  events,  34  were  fully  satisfied 
while  another  2  were  partially  satisfied.  For  the  remaining  8  key  processes, 
4  were  still  in  progress,  and  4  processes  concerning  correcting  defects 
found  were  “not  applicable”  because  initial  testing  results  had  not  yet 
disclosed  Year  2000  defects.  However,  some  of  the  test  results  that  were 
obtained  during  our  review  were  still  being  analyzed  by  DOD. 


Table  3:  Summary  of  Test  Events  Satisfying  GAO  Key  Processes 

Selected  functional  area  test  event 

Fully  satisfied 

Partially  satisfied 

In  progress 

N/A 

Total 

Health  Affairs 

8 

1 

1 

1 

11 

Communications 

9 

0 

1 

1 

11 

Personnel 

9 

0 

1 

1 

11 

Logistics 

8 

1 

1 

1 

11 

Total 

34 

2 

4 

4 

44 

Note:  Due  to  differences  in  scope  and  complexity  of  the  test  events,  the  results  of  individual  functions 
are  not  comparable. 

In  all  cases  where  we  determined  that  the  test  events’  key  processes  called 
for  in  our  guide  had  only  been  partially  satisfied,  the  PSAs  and  test 
managers  agreed  to  address  our  concerns  and  initiate  corrective  actions. 
For  example: 

•  While  Health  Affairs  prepared  procedures  for  its  test  event,  these 
procedures  were  not  sufficiently  detailed  and  did  not  define  each  step  to 
be  executed  or  precisely  define  input  data.  As  a  result,  it  was  necessary 
for  system  operators  to  augment  the  test  procedures  during  the  test’s 
execution.  While  this  approach  was  satisfactorily  carried  out  because 
the  relative  simplicity  of  the  test  event  permitted  face-to-face 
coordination  and  synchronization  of  the  procedures,  it  was 
unnecessarily  risky  and  could  have  been  easily  avoided  by  ensuring  that 
test  procedures  were  complete.  Health  Affairs  officials  agreed  that  more 
detailed  procedures  should  have  been  established,  and  they  committed 
to  ensuring  that  other  Health  Affairs  test  events  have  them. 


Page  9 


GAO/AIMD-OO-12  DOD  V2K  End-to-End  Testing 


B-283564 


•  Although  the  Logistics  function  is  reliant  on  telecommunications 
providers  such  as  military  installations  and  the  Defense  Information 
Systems  Agency  (DISA),  at  the  time  of  the  test  event  we  observed, 
documentation  offering  assurances  that  installations’ 
telecommunications  infrastructures  were  Y2K  compliant  was  not 
provided  by  Logistics  functional  managers.  Our  test  guide  states  that,  in 
order  to  ensure  that  all  systems  in  the  chain  of  support  function  as 
intended,  the  telecommunications  infrastructure  that  interconnects  the 
systems  must  be  compliant  and  ready  for  testing.  Subsequent  to  our 
review,  Logistics  officials  provided  information  showing  that 
installations’  telecommunications  infrastructures  had  been  included  in 
installation  test  events.  Logistics  officials  agreed,  however,  that  they  had 
not  yet  confirmed  the  Y2K  compliance  of  the  infrastructures,  and 
reported  that  they  have  subsequently  initiated  steps  to  do  so. 


Conclusions 


Given  that  virtually  all  Defense  business  functions  and  military  operations 
rely  heavily  on  technology,  it  is  vital  that  Year  2000  end-to-end  testing 
efforts  be  effectively  planned  and  executed.  All  four  of  the  individual  test 
events  that  we  reviewed  were  well-managed  because  each  either  satisfied 
or  had  steps  underway  or  planned  to  address  all  relevant  end-to-end 
management  key  processes  specified  in  our  test  guide.  Moreover, 
differences  between  the  functional  areas’  approaches  to  implementing 
these  key  processes  were  generally  commensurate  with  the  events’  scope 
and  complexity.  Finally,  reported  functional  area  status  information 
indicates  that  end-to-end  tests  are  generally  progressing  on  schedule. 
However,  DOD  does  not  yet  have  assurance  that  all  of  its  communications 
systems  will  be  Year  2000  compliant  and,  as  such,  should  ensure  that  all 
mission-critical  communications  systems  are  tested. 


Recommendation  We  recommen<3  that  the  Secretary  of  Defense  direct  the  Senior  Civilian 

Official  of  the  Office  of  the  Assistant  Secretary  of  Defense  for  Command, 
Control,  Communications,  and  Intelligence  to  report  to  the  Deputy 
Secretary  immediately  on  plans  for  end-to-end  testing  the  31  mission- 
critical  communications  systems,  including  milestones  for  executing  tests 
and  reporting  test  results,  or  to  otherwise  justify  in  writing  to  the  Deputy 
Secretary  why  any  of  the  systems  will  not  be  included  in  an  end-to-end  test 
event. 
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Agency  Comments  and 
Our  Evaluation 


DOD  concurred  with  our  findings  and  partially  concurred  with  our 
recommendation  to  report  to  the  Deputy  Secretary  on  the  status  and  plans 
for  Y2K  testing  of  the  31  mission-critical  communications  systems 
disclosed  in  our  report. 


In  partially  concurring  on  the  recommendation,  DOD  stated  that  during  the 
July  through  August  1999  period  of  our  review,  testing  data  in  the  OSD  Y2K 
database  was  still  evolving,  and  as  a  result,  test  data  were  incomplete  for 
many  of  the  31  systems.  Since  then,  resolution  has  been  reached  on  the 
testing  status  of  the  31  communications  systems.  DOD  reported  and 
provided  documentation  to  show  that  (1)  Y2K  testing  for  14  of  the  31 
systems  has  been  completed,  (2)  9  systems  do  not  process  dates  and  are 
exempt  from  end-to-end  test  requirements,  (3)  4  systems  are  trusted 
systems,  which  cannot  be  tested  in  a  Y2K  environment  due  to  safety, 
security,  or  operational  necessity  reasons,  (4)  2  systems  are  developmental 
systems  that  will  not  be  deployed  before  the  millennium  rollover,  (5)  1 
system  has  been  reclassified  as  a  nonmission-critical  system  and  does  not 
require  additional  testing,  and  (6)  1  system  is  scheduled  to  complete  testing 
by  October  15,  1999.  We  have  not  verified  the  status  information  provided 
by  DOD. 


We  are  sending  copies  of  this  report  to  Representative  John  P.  Murtha, 
Ranking  Minority  Member,  Subcommittee  on  Defense,  House 
Appropriations  Committee;  Senator  John  Warner,  Chairman,  and  Senator 
Carl  Levin,  Ranking  Minority  Member,  Senate  Committee  on  Armed 
Services;  Senator  Ted  Stevens,  Chairman,  and  Senator  Daniel  Inouye, 
Ranking  Minority  Member,  Subcommittee  on  Defense,  Senate  Committee 
on  Appropriations;  and  Representative  Floyd  Spence,  Chairman,  and  Ike 
Skelton,  Ranking  Minority  Member,  House  Committee  on  Armed  Services. 

We  are  also  sending  copies  to  the  Honorable  John  Koskinen,  Chair  of  the 
President’s  Year  2000  Conversion  Council;  the  Honorable  William  Cohen, 
Secretary  of  Defense;  the  Honorable  John  Hamre,  Deputy  Secretary  of 
Defense;  General  Henry  Shelton,  Chairman  of  the  Joint  Chiefs  of  Staff; 
Arthur  Money,  Senior  Civilian  Official  of  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Command,  Control,  Communications,  and 
Intelligence;  and  the  Honorable  Jacob  Lew,  Director  of  the  Office  of 
Management  and  Budget.  Copies  will  also  be  made  available  to  others  upon 
request. 
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Should  you  or  your  staff  have  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-6240. 1  can  also  be  reached  by  e-mail  at 
brockj.aimd@gao.gov.  Other  points  of  contact  and  key  contributors  to  this 
report  are  listed  in  appendix  IV. 

Sincerely  yours, 


Jack  L.  Brock,  Jr. 

Director,  Governmentwide  and  Defense 
Information  Systems 


GAO/AIMD-OO- 1 2  DOD  Y2K  End-to-End  Testing 


Page  13 


GAO/AIMD-OO-12  DOD  Y2K  End-to-End  Testing 


Appendix  I _ __ _ 

Briefing  on  DOD  Y2K  Functional  End-to-End 
Testing:  Progress  and  Test  Event  Management 


DOD  Y2K  Functional  End-to-End 
Testing:  Progress  and  Test  Event 

Management 


House  Appropriations  Committee 
September  14, 1999 
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Briefing  on  DOD  Y2K  Functional  End-to-End 
Testing:  Progress  and  Test  Event 
Management 


i 

^  GAO 

^^^^^^?^AccountablHty  *  Integrity  *  Reliability 


Briefing  Overview 


•  Introduction 

•  Objectives 

•  Scope  and  Methodology 

•  Background 

•  Objective  1:  Status  and  Progress  Information  Summary 

•  Objective  2:  Results  of  Specific  Test  Events  Reviewed 

•  Findings 

•  Conclusions 

•  Recommendation 
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i 

G  A  O 

Accountability  *  Integrity  *  Reliability 


Introduction 


•  In  August  1998,  the  Deputy  Secretary  of  Defense  directed 
the  Principal  Staff  Assistants  (PSAs),  who  are  responsible 
for  Defense’s  functional  areas,  to  coordinate,  facilitate  and 
monitor  a  series  of  Year  2000  functional  evaluations  (Y2K 
functional  testing)  to  ensure  functions  will  operate  properly 
in  a  Y2K  environment. 

•  Y2K  functional  testing  is  one  of  three  Defense  Y2K  end- 
to-end  test  and  evaluation  efforts.  Y2K  end-to-end  testing 
is  testing  performed  to  verify  that  systems  collectively 
supporting  an  organizational  core  business  function  or 
operation  interoperate  as  intended  in  a  Y2K  environment. 

•  The  PSAs  core  business  processes  are  referred  to  as 
“threads.” 


3 


Page  16 


GAO/AIMD-OO- 1 2  DOD  Y2K  End-to-End  Testing 


Appendix  I 

Briefing  on  DOD  Y2K  Functional  End-to-End 
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i 

iGAO 

Accountability  *  Integrity  *  Reliability 


GAO  Objectives 


•  At  the  request  of  the  Chairman,  House  Appropriations 
Committee,  Defense  Subcommittee,  we  initiated  a 
review  of  the  effectiveness  of  DOD  Y2K  functional  end- 
to-end  test  management.  The  objectives  of  the  review 
are  to: 


•  Assess  status/progress  on  all  test  events  within  four 
functional  areas:  Health  Affairs,  Communications, 
Personnel,  and  Logistics 

•  Review  the  management  effectiveness  of  a  critical 
test  event  for  each  of  the  four  functional  areas 
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i 

G  A  0 
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Scope  and  Methodology 


•  The  test  events  we  reviewed  were: 

•  Health  Affairs 

Patient  Care/Issuance  and  Processing  of  Blood 
Requests 

•  Communications 

Joint  User  Switch  Exercise  (JUSE-99-Y2K) 

•  Personnel 

Army  Personnel/Mobilization/Reserve  Unit 

•  Logistics 

Joint  tests  of  requisition  and  receipt  processes 
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•  For  all  test  events,  we  compared  reported  status 
information  to  test  plan  milestones  to  identify  variances. 


•  For  the  selected  test  events,  we: 

•  observed  a  part  of  the  test  event  for  each  functional 
area  and 

•  compared  event  management  activities  to  the  key 
processes  for  end-to-end  testing  as  specified  in  our 
Year  2000  Test  Guide. 

•  The  test  events  we  reviewed  were  selected  based  on  the 
PSA’s  designation  that  the  events  were  of  key  importance 
to  the  business  function. 
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Briefing  on  DOD  Y2K  Functional  End-to-End 
Testing:  Progress  and  Test  Event 
Management 


i 

1GAQ 

Accountability  *  Integrity  +  Reliability 


Scope  and  Methodology 


•  The  test  events  we  reviewed  assessed  calendar  year 
rollover  only. 


•  We  briefed  the  respective  PSAs  and  test  event 
managers  on  the  results  of  our  reviews  following  the 
conclusion  of  each  event. 


•  Health  Affairs,  June  15,  1999 

•  Communications,  July  12,  1999 

•  Personnel,  July  15,  1999 

•  Logistics,  August  17,  1999 
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Testing:  Progress  and  Test  Event 
Management 


•  In  commenting  on  the  briefings,  PSAs  and  test  managers  agreed 
with  the  results.  This  briefing  consolidates  the  results  of  those 
individual  briefings,  and  provides  updated  status  information  on 
each  functional  area’s  testing  progress.  We  also  clarified  the 
results  of  our  analysis  of  the  Logistics  test  event  based  on 
additional  information  provided  by  Logistics  officials  after  our 
September  14, 1999  briefing. 

•  Due  to  differences  in  scope  and  complexity,  the  results  of  our 
review  of  the  test  events  are  not  comparable  across  functions. 


•  We  performed  our  audit  work  in  accordance  with  generally 
accepted  government  auditing  standards  between  March  and 
September  1999. 
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Background 


•  DOD  requires  PSAs  to  verify  that  their  respective  critical 
business  processes  can  function  properly  in  a  Y2K 
environment. 


•  PSAs  are  to: 

•  document  mission-critical  functions  and  systems 
supporting  those  functions, 

•  coordinate,  facilitate,  and  monitor  Y2K  end-to-end  test 
and  evaluation  activities  of  services,  agencies,  and 
commands,  and 

•  in  some  cases,  conduct  Y2K  end-to-end  functional 
testing. 
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Background 


•  The  scope  and  complexity  of  the  test  events  varied  in 
terms  of,  for  example,  the  numbers  of  systems  tested  and 
locations  involved. 

•  Health  Affairs  test  event  involved  a  two  system  cluster 
in  a  simulated  Y2K  environment.  In  contrast,  the 
Logistics  test  event  included  6  operational  systems  and 
multiple  geographical  dispersed  installations. 

•  Personnel  test  event  involved  only  Army  systems. 
Communications,  Logistics  and  Health  Affairs  test 
events  involved  more  than  one  Defense  component. 
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i 

4  GAO 

Accountability  *  Integrity  *  Reliability 


Background 


•  Army  Personnel  and  Logistics  chose  different  methods  for 
testing  threads  that  spanned  more  than  one  day. 

•  Army  Personnel  divided  its  threads  into  what  are 
termed  “pairs”  and  tested  transactions  across  each 
“pair”  of  systems  using  the  critical  rollover  date. 


•  Logistics  used  a  5-day  test  cycle  that  commences  on 
the  day  before  rollover  (i.e.,  9/30/1999,  12/31/1999,  and 
2/28/2000).  This  strategy  verifies  the  mission-critical 
thread  across  the  date  rollover,  rather  than  verify  all 
systems  ability  to  correctly  perform  their  respective 
function  across  the  date  rollover. 
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i 

GAO 

Accountability  *  integrity  +  Reliability 


Objective  1 :  Status  and 
Progress  Information  Summary 


•  Criteria:  Management  should  collect  timely,  reliable  test 
progress  and  status  information  in  order  to  ensure  that 
testing  is  progressing  according  to  plans  and  that 
problems  are  resolved  and  testing  requirements  are  met. 

•  Finding:  Health  Affairs,  Personnel,  and  Logistics  are 
collecting  the  progress  and  status  information. 

•  Finding:  At  the  time  of  our  July  1 999  briefing  to  the  PSA, 
Communications  was  unable  to  provide  us  with  complete 
information  on  functional  test  event’s  progress  for  all 
mission-critical  systems  it  had  identified.  Subsequently, 
additional  information  was  provided  that  shows  the  status 
of  all  systems;  however,  Communications  reported  that 
tests  have  not  yet  been  planned  for  31  of  the  systems. 
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Objective  1 :  Status  and 
Progress  Information  Summary 

(continued) 


•  Finding:  With  the  exception  of  the  tests  that  need  to  be 
planned  for  the  31  Communications  systems,  the 
available  test  progress  information  shows  that  functional 
areas  end-to-end  test  events  are  on  schedule. 
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^^^^“■^Accountablllty  *  Integrity  *  Reliability 


Objective  1 :  Status  and 
Progress  Information 


Health  Affairs 


Test  Type 

Reported  Status  of  Testing  Efforts 

•  Patient  Care 

26  out  of  26  threads  have  been  tested. 

Testing  was  completed  on  schedule. 

•  Patient 

Administration 

30  out  of  30  threads  have  been  tested. 

Testing  was  completed  on  schedule. 

•  Medical 
Logistics 

26  out  of  55  threads  have  been  tested. 

Testing  commenced  on  July  2, 1999  and  is  on  schedule 
to  be  completed  by  the  end  of  September  1999. 
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Objective  1 :  Status  and 
Progress  Information  (continued) 


Communication 


Sest  Type 

Reported  Status  of  Testing  Events 

•  Army 

Of  68  systems: 

19  have  been  tested,  4  have  not  been  planned  and  are  late,  and  45  systems 
do  not  require  tests.1 

•  Air  Force 

Of  32  systems: 

10  have  been  tested,  14  have  not  been  planned  and  are  late,  and  8  systems 
do  not  require  tests.1 

•  Navy 

Of  123  systems: 

30  have  been  tested,  4  have  not  been  planned  and  are  late,  and  89  systems 
do  not  require  tests.1 

•  Marine 
Corps 

Of  6  systems: 

1  has  been  tested,  1  has  not  been  planned  and  is  late,  and  4  systems  do  not 
require  tests.1 

•  CINCs 

Of  15  systems:  * 

5  have  been  tested,  and  8  have  not  been  planned  and  are  late,  and  2 
systems  do  not  require  tests.1 

•  DISA 

Of  19  systems: 

12  have  been  tested  and  7  systems  do  not  require  tests.1 

'Communications  relied  on  the  Y2K  Office  database  to  identify  systems  that  do  not  require  testing.  Systems  that  do  not  require  tests  are:  1)  developmental  systems,  2)  do  not  process 
dates,  or  3)  arc  stand  alone  systems.  _ 
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Personnel _ 

Test  Type  Reported  Status  of  Testing  Efforts 

•  Army  19  out  of  19  threads  have  been  tested. 

Testing  was  completed  on  schedule. 

•  Navy  Testing  is  in  progress,  0  out  of  18  threads 

are  complete.  The  testing  was  started  ahead 
of  schedule  and  is  planned  to  be  completed 
_ by  October  17, 1999. _ 

•  Marine  7  out  of  7  threads  have  been  tested. 

Corps  The  testing  began  on  time;  however,  1  thread 
was  not  completed  until  September  9,  1999, 
_ instead  of  August  20, 1 999. _ 

•  Air  Force  10  out  of  1 0  threads  have  been  tested. 

Testing  was  completed  on  schedule. 

•  Civilian 

•  DEERS/ 

RAPIDS2 

2Defcnsc  Enrollment  Eligibility  Reporting  Systems  (DEERS),  Real  Time  Automated  Personnel  ID  Card  System  (RAPIDS) 


2  out  of  2  threads  have  been  tested. 
Testing  was  completed  on  schedule. 

4  out  of  4  threads  have  been  tested. 
Testing  was  completed  on  schedule. 
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Objective  1 :  Status  and 
Progress  Information  (continued) 


Logistics 


Test  Type 

Reported  Status  of  Testing  Efforts 

•  Intra-component 

All  components  have  completed  testing. 

•  Inter-component 
(Requisition) 

41  of  41  threads  have  been  tested. 

Testing  was  completed  on  schedule. 

•  Inter-component 
(Shipment) 

4  of  4  threads  have  been  tested. 

Testing  was  completed  on  schedule. 

•  Inter-component 
(Receipt) 

4  of  4  threads  have  been  tested. 

Testing  was  completed  on  schedule. 

•  Inter-component 
(Inventory  Control  & 
Asset  Status) 

4  of  4  threads  have  been  tested. 

Testing  was  completed  on  schedule. 
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Objective  2:  Results  of  Specific 
Test  Events  Reviewed 


Satisfied 


In  Progress 


Partially 

Satisfied 


o 

Not 

Applicable 


GAO  Test  Criteria 

Logistics 

Define  the  system  boundaries  of  the  end-to-end 
test(s) 

• 

• 

• 

■;  ■ 

Secure  the  commitment  of  data  exchange  partners 

• 

• 

• 

• 

Establish  an  inter-organizational  test  team 

• 

• 

• 

• 

Confirm  Y2K  compliance  of  telecommunications 

• 

• 

• 

# 

Schedule  and  plan  end-to-end  test(s) 

• 

• 

• 

Prepare  end-to-end  procedures  and  data 

• 

• 

• 

• 

Define  end-to-end  test  exit  criteria 

• 

• 

• 

• 

Execute  end-to-end  test(s) 

• 

• 

• 

• 

Document  end-to-end  test  results 

• 

• 

• 

• 

Correct  Y2K  defects 

o 

o 

o 

o 

Ensure  that  end-to-end  test  exit  criteria  are  met 

0 

0 

0 

Note:  Due  to  differences  in  scope  and  complexity  of  the  test  events,  the  results  of  individual  functions  are  not  comparable. 


JThe  functional  areas  were  abbreviated:  Health  Affairs  (HA),  Communications  (Comm),  and  Personnel  (Pers) 
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Finding:  End-to-end  test 
boundaries  defined 


•  Criteria:  Organizations  need  to  assess  their  mission-critical 
business  functions  and  dependencies  and,  on  the  basis  of 
this  assessment,  define  boundaries  for  end-to-end  tests. 

•  Finding:  All  of  the  functional  areas  defined  test  boundaries 
for  the  test  events  we  witnessed.  For  example, 
Communications  defined  boundaries,  including  geographic 
sites,  deployed  forces,  satellites,  and  control  centers. 

Health  Affairs  identified  three  critical  functions:  Patient  Care, 
Patient  Administration,  and  Medical  Logistics.  Patient  Care 
was  broken  down  into  mission-critical  threads,  one  of  which 
was  issuing  and  processing  blood  requests. 
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Finding:  Commitment  of  key 
data  exchange  partners 
secured 


•  Criteria:  Because  end-to-end  testing  addresses  business 
areas  or  functions  that  involve  multiple  organizations, 
participation  of  all  key  business  area  data  exchange 
partners  should  be  secured. 

•  Finding:  All  of  the  functional  areas  secured  the 
commitment  of  key  data  exchange  partners.  For  example, 
Communications  data  exchange  partners  from  different 
parts  of  the  world  agreed  to  send  and  receive  data, 
including  Ft.  Monmouth,  St.  Louis,  Atlantic  Command,  and 
Strategic  Command  for  the  test  that  we  reviewed. 
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i 
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Findings:  Interorganizational 
end-to-end  teams  established 


•  Criteria:  A  team  composed  of  representatives  from  each 
of  the  organizations  participating  in  the  test  should  be 
formed  to  manage  the  planning,  execution,  and  reporting 
of  the  test. 


•  Finding:  All  functional  areas  established  an  inter¬ 
organizational  team  that  planned,  executed,  and  reported 
on  the  test  event.  For  example,  Health  Affairs  established 
a  team  that  included  the  TRICARE  Management  Activity 
Office,  IV&V  contractors,  and  technical  support  staff  from 
Advanced  Technology  Integration  Center. 

•  Finding:  Logistics  interorganizational  team  included 
relevant  data  exchange  partners,  such  as  the  military 
services  and  Defense  Logistics  Agency. 
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Findings:  Y2K  compliance  of 
telecommunications  partially 
ensured 


•  Criteria:  In  order  to  ensure  that  all  systems  in  the  chain  of 
support  function  as  intended,  the  telecommunications 
infrastructure  that  interconnects  the  systems  must  be 
compliant  and  ready  for  testing. 

•  Finding:  Health  Affairs  ensured  that  its 
telecommunications  infrastructure  for  the  test  (i.e.,  local 
area  network)  was  Y2K  compliant. 

•  Finding:  Communications,  Personnel,  and  Logistics  end- 

to-end  test  events  used  the  Unclassified  (but  Sensitive) 
Internet  Protocol  Router  Network  (NIPRNET).  Defense 
Information  Systems  Agency  reported  that  the  NIPRNET 
has  been  certified  Y2K  compliant  and  is  fully 
implemented.  _ 
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Findings:  Y2K  compliance  of 
telecommunications  partially 
ensured  (continued) 


•  Finding:  Communications  and  Personnel  have  verified 
that  their  local  telecommunications  infrastructure  are  Y2K  . 
compliant.  Logistics  had  not  verified  that  its  local 
telecommunications  infrastructure  is  Y2K  compliant,  but 
has  since  initiated  steps  to  do  so. 


23 


Page  36 


GAO/AIMD-OO- 1 2  DOD  Y2K  End-to-End  Testing 


Appendix  I 

Briefing  on  DOD  Y2K  Functional  End-to-End 
Testing:  Progress  and  Test  Event 
Management 


^  G AO _ 

Accountability  +  Integrity  *  Reliability 


Finding:  End-to-end  tests 
planned  and  scheduled 


•  Criteria:  A  plan  should  be  developed  that  defines  key 
tasks  and  requirements  in  preparing  for,  executing,  and 
documenting  results  of  testing.  Responsibility  for 
performing  key  tasks  and  quality  assurance/independent 
Verification  and  Validation  (IV&V)  need  to  be  established. 

•  Finding:  All  functional  areas  established  plans  that 

identified  key  tasks,  schedules,  roles  and  responsibilities 
for  the  participants  in  the  test  events.  For  example, 
detailed  plans  and  schedules  were  provided  in  the  Health 
Affairs  Master  Test  Plan  and  Patient  Care  Event  Plan, 
which  addressed  the  IV&V  contractor’s  roles  and 
responsibilities.  Also,  Personnel  had  an  Army  Master 
Test  Plan  and  Army  System  Test  Plans  that  identified  key 
tasks,  schedules,  roles  and  responsibilities. _ 
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Findings:  End-to-end  test 
procedures  prepared 


•  Criteria:  Test  procedures  and  data  should  include  steps, 
cases,  and  input  conditions  that  verify  the  correct  handling 
of  critical  dates. 


•  Finding:  Most  functional  areas  prepared  test  procedures 
and  data.  For  example,  Army  Personnel  prepared  baseline 
input  test  data  and  generated  baseline  output.  Army 
Personnel  then  aged  the  baseline  test  data  inputs  and 
outputs  to  reflect  a  Y2K  environment  and  used  these  to 
verify  correct  handling  of  the  calendar  year  rollover  date. 

•  Finding:  Health  Affairs  developed  high-level  test 
procedures;  however,  it  was  necessary  for  operators  to 
augment  the  test  procedures  to  complete  the  test.  Health 
Affairs  officials  agreed  that  more  detailed  procedures 
should  have  been  established. 
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Finding:  Test  exit  criteria 
defined 


•  Criteria:  The  conditions  or  requirements  for  successfully 
completing  end-to-end  testing  need  to  be  established. 

•  Finding:  All  functional  areas  defined  test  exit  criteria  for 
the  specific  test  events.  For  example,  Communications 
specified  that  the  test  exit  criteria  for  successfully 
completing  voice  communications  were  to  hear  a  person 
at  the  other  end  of  the  line  with  a  specified  clarity.  It 
specified  that  the  test  exit  criteria  for  successfully 
completing  data  communications  were  for  all  the  message 
recipients  to  receive  the  “test”  message  with  all 
attachments  over  a  72-hour  period. 
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Finding:  End-to-end  test  plans 
were  followed 


•  Criteria:  Tests  should  be  executed  in  accordance  with 
established  plans  and  procedures. 

•  Finding:  All  test  events  adhered  to  the  established  plans 
and  procedures.  For  instance,  the  Army  Personnel  test 
event  was  executed  in  accordance  with  both  the  master 
and  system  test  plans.  Similarly,  the  Logistics  test  event 
was  executed  in  accordance  with  the  test  plans  and 
procedures. 
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Finding:  Test  results  were 
documented 


•  Criteria:  End-to-end  test  results  should  be  documented  so 
that  the  data  can  be  used  to  validate  that  test  exit  criteria 
have  been  met  and  to  assess  and  correct  problems 
discovered  during  testing. 

•  Finding:  All  functional  areas  documented  test  results. 

Both  Health  Affairs  and  Personnel  documented  results  by 
printing  out  computer  screen  information.  Logistics  and 
Communications  documented  test  results  on  a  results 
reporting  form. 
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Finding:  No  Y2K  defects  were 
found  during  the  test  event 


•  Criteria:  On  the  basis  of  interorganization  specified 
criteria,  such  as  defect  severity  and  test  exit  criteria, 
defects  identified  during  the  test  should  be  prioritized  and 
corrected. 


•  Finding:  Initial  results  from  the  part  of  the  test  event  that 
we  observed  indicated  that  none  of  the  functional  areas 
experienced  Y2K  defects.  However,  analysis  of  the  test 
results  has  not  yet  been  completed  by  all  of  the  functional 
area  data  analysis  centers. 
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Finding:  Effort  to  ensure 
end-to-end  test  meets  exit 
criteria  is  ongoing 


•  Criteria:  Test  results  should  be  compared  to  test  exit 
criteria  to  ensure  that  specified  conditions  are  met. 


•  Finding:  All  functional  areas  are  still  reviewing  test  results 
at  the  data  collection  centers.  For  example,  Logistics’ 
Exercise  Operation  Center  is  collecting  results  reporting 
forms  and  assessing  the  satisfaction  of  test  exit  criteria. 

In  addition,  Communications-Electronics  Command  is 
collecting  test  results  for  analysis  and  determination  of 
test  exit  criteria  satisfaction. 
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Conclusions 


•  Three  of  the  four  PSAs  were  able  to  provide  us  with 
information  needed  to  effectively  track  the  progress  and 
status  of  their  respective  inventories  of  end-to-end  test 
events.  Communications  was  not  initially  able  to  provide 
this  information  on  all  its  test  events.  While  additional 
status  information  was  provided  recently,  this  information 
identified  mission-critical  systems  for  which  end-to-end 
testing  had  not  yet  been  planned,  thus  leaving  very  little 
time  to  ensure  that  they  are  included  in  an  end-to-end  test 
event. 

•  The  four  test  events  that  we  reviewed  were  effectively 
managed  based  on  the  satisfaction  of  most,  if  not  all,  of 
the  key  end-to-end  testing  processes  in  our  guidance. 
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Conclusions 


•  The  Logistics  test  event  was  somewhat  limited  by  not 
ensuring  the  Year  2000  compliance  of  the  end-to-end 
partners’  local  telecommunications  equipment  connecting 
the  partners  to  the  NIPRNET.  Logistics  has  initiated 
appropriate  steps  to  do  so. 

•  The  test  events’  respective  approaches  and  the  degree  of 
test  planning,  execution,  and  reporting  on  the  four  test 
events  varied.  However,  the  variances  were  generally 
commensurate  with  the  unique  characteristics  of  the 
respective  events.  Such  variability  in  test  approaches  is 
consistent  with  our  test  guidance. 
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Recommendation 


•  We  recommend  that  the  Secretary  of  Defense  direct  the 
Senior  Civilian  Official  of  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Command,  Control 
Communications,  and  Intelligence  to  report  to  the  Deputy 
Secretary  immediately  on  plans  for  end-to-end  testing  the 
31  systems,  including  milestones  for  executing  tests  and 
reporting  test  results,  or  to  otherwise  justify  in  writing  to 
the  Deputy  Secretary  why  any  of  the  systems  will  not  be 
included  in  an  end-to-end  test  event. 


33 


Page  46 


GAO/AIMD -00-12  DOD  Y2K  End-to-End  Testing 


Appendix  II 


Objectives,  Scope,  and  Methodology 


As  requested  by  the  House  Committee  on  Appropriations,  Subcommittee 
on  Defense,  our  objectives  were  to  (1)  assess  the  status  and  progress  of  all 
test  events  within  four  functional  areas — Health  Affairs,  Communications, 
Personnel,  and  Logistics — and  (2)  review  the  management  effectiveness  of 
a  critical  test  event  for  each  of  the  four  functional  areas.  Together,  these 
functional  areas  are  performing  thousands  of  end-to-end  tests  to  ensure 
that  key  business  processes  and  systems  can  continue  operating  into  the 
year  2000. 

To  meet  our  first  objective,  we  obtained  status  and  progress  information 
for  the  aforementioned  functional  areas  and  compared  the  reported  status 
information  to  milestones  contained  in  individual  functional  test  plans1  to 
identify  variances.  We  discussed  this  information  with  DOD  officials  and 
personnel  from  the  Office  of  the  Assistant  Secretary  of  Defense  (Health 
Affairs) ,  the  Office  of  the  Assistant  Secretary  of  Defense  for  Command, 
Control,  Communications,  and  Intelligence,  the  Office  of  the 
Undersecretary  of  Defense  (Personnel  and  Readiness) ,  and  the  Deputy 
Undersecretary  of  Defense  (Logistics).  Also,  for  each  of  the  functional 
areas,  where  necessary,  we  obtained  updated  status  and  progress 
information  on  end-to-end  test  events. 

To  meet  our  second  objective,  we  selected  one  specific  test  event  for  each 
functional  area.  The  four  selected  test  events  were  based  on  each  PSA’s 
designation  that  the  test  event  was  of  key  importance  in  ensuring  that  the 
function  could  continue  unaffected  at  and  after  the  turn  of  the  century.  The 
selected  test  events,  and  the  dates  and  locations  we  observed  the  events, 
were 

•  Health  Affairs — Patient  Care/Issuance  and  Processing  of  Blood 
Requests  (May  18, 1999)  at  the  Advanced  Technology  Integration  Center 
in  Falls  Church,  Virginia. 

•  Communications — Joint  User  Switch  Exercise  (JUSE-99-Y2K) 

(June  10,  1999)  at  the  Army  Communications-Electronics  Command, 
Fort  Monmouth,  New  Jersey. 

•  Personnel — Army  Personnel/Mobilization/Reserve  Unit  (June  17, 1999) 
at  the  Army  Personnel  Command,  Alexandria,  Virginia. 


‘Updated  plans  included  in  our  review  were  the  December  15, 1998,  plan  for 
communications;  the  January  1999,  plan  for  health  affairs;  the  January  31, 1999,  plan  for 
logistics;  and  the  January  28, 1999,  plan  for  personnel. 
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•  Logistics — Intercomponent  test  of  requisition  and  receipt  processes 
(June  24-25, 1999)  at  the  Navy  Fleet  Material  Support  Office  in 
Mechanicsburg,  Pennsylvania. 

For  the  selected  test  events,  we  interviewed  DOD  officials  and  reviewed 
pertinent  documentation  for  each  event,  including  test  event  plans,2 
procedures,  conditions,  exit  criteria,  results,  reports,  defects,  correction 
action  plans,  and  we  observed  the  actual  execution  of  the  test  event.  We 
then  compared  the  particulars  of  each  event  to  our  Year  2000  test  guide’s 
end-to-end  testing  key  processes,  identified  variances,  and  discussed  with 
test  officials  the  reasons  for  and  impacts  of  any  variances. 

To  supplement  our  documentation  reviews  and  observations,  we 
interviewed  DOD  officials,  including  those  from  the  TRICARE  Military 
Health  Systems,  the  Army  Communications-Electronics  Command,  the 
Office  of  the  Deputy  Chief  of  Staff  for  Personnel,  and  the  Office  of  the 
Deputy  Undersecretary  of  Defense  (Logistics) ;  test  event  coordinators;  and 
test  directors  regarding  additional  clarifications  after  our  visit.  These 
officials  addressed  telecommunications  infrastructure  Year  2000 
compliance  issues.  They  also  provided  additional  documentation  from  the 
test  event  we  witnessed  (i.e.,  test  results,  quick  look,  and  final  reports). 
Due  to  the  time  criticality  of  the  year  2000,  as  our  reviews  were  completed 
on  each  of  the  functional  areas,  we  provided  briefings  detailing  our 
observations  to  each  Defense  PSA  and  test  director  as  follows: 

•  Health  Affairs — June  15, 1999, 

•  Communications — July  12, 1999, 

•  Personnel — July  15,  1999,  and 

•  Logistics — August  17,  1999. 

We  performed  our  audit  work  primarily  at  DOD  headquarters  and  at  the 
test  event  locations  described  above.  We  requested  and  received  comments 
on  a  draft  of  this  report  from  DOD  and  incorporated  those  comments  as 
appropriate.  We  performed  our  audit  work  from  March  through  September 
1999  in  accordance  with  generally  accepted  government  auditing 
standards. 


2Test  event  plans  included  in  our  review  were:  Military  Health  Systems  Patient  Care 
Functional  Readiness  Assessment  Test  Plan,  vl.2,  April  1999;  Communications  Joint  User 
Switch  Exercise  99-Y2K  Exercise  Directive,  April  1999;  Army  Personnel  Command  Test 
Plan,  June  1999;  Army  System  Test  Plans,  June  1999,  for  EDAS,  SIDPERS-3,  and  PEPDUS; 
and  Logistics  Exercise  Directive,  May  1999. 
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COMMAND.  CONTROL, 
COMMUNICATIONS,  AND 
INTELLIGENCE 


OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
6000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-6000 


October  05,  1999 


Mr.  Jeffrey  C.  Steinhoff 
Acting  Assistant  Comptroller  General 
Accounting  and  Information  Management 
Division 

U.S.  General  Accounting  Office 
Washington,  DC  20548 

Dear  Mr.  Steinhoff: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  draft  report,  GAO/AIMD-99-288,  "DEFENSE  COMPUTERS:  DoD  Y2K  Functional 
End-to-End  Testing  Progress  and  Test  Management,”  dated  September  17,  1999,  (GAO  Code 
51 1659/OSD  Case  1896).  The  Department  acknowledges  receipt  of  this  report. 

The  Department  of  Defense  concurs  with  the  Draft  GAO  Report  and  partially  concurs  with 
the  recommendation.  The  report  identifies  testing  issues  associated  with  31  communications 
systems.  During  the  investigation  period,  July  1999  and  early  August  1999,  testing  data  in  the . 
OSD  Y2K  database  was  still  evolving.  This  resulted  in  incomplete  test  data  for  many  of  the  31 
systems  identified  in  this  report.  The  referenced  DoD  communication  systems  status  has  been 
resolved  and  no  additional  test  events  beyond  those  listed  are  necessary. 

The  DoD  appreciates  the  opportunity  to  comment  on  the  draft  report.  Technical  comments 
were  provided  under  separate  cover. 


Sincerely, 


Enclosure 
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GAO  DRAFT  REPORT  DATED  SEPTEMBER  17, 1999 
(GAO  CODE  511659)  OSD  CASE  1896 


SUBJECT:  " DEFENSE  COMPUTERS:  DOD  Y2K  FUNCTIONAL  END-TO-END 
TESTING  PROGRESS  AND  TEST  EVENT  MANAGEMENT” 

GAO  Recommendation:  We  recommend  that  the  Secretary  of  Defense  direct  the  Senior 
Civilian  Official  of  the  Office  of  the  Assistant  Secretary  of  Defense  (Command,  Control, 
Communications,  and  Intelligence)  to  report  to  the  Deputy  Secretary  immediately  on  plans  for 
end-to-end  testing  the  31  mission-critical  communications  systems,  including  milestones  for 
executing  tests  and  reporting  test  results,  or  to  otherwise  justify  in  writing  to  the  Deputy 
Secretary  why  any  of  the  systems  will  not  be  included  in  an  end-to-end  test  event. 

DoD  Response:  The  Department  of  Defense  concurs  with  the  Draft  GAO  Report  and  partially 
concurs  with  the  recommendation.  The  report  identifies  testing  issues  associated  with  31 
communications  systems.  During  the  investigation  period,  July  1999  and  early  August  1999, 
testing  data  in  the  OSD  Y2K  database  was  still  evolving.  This  resulted  in  incomplete  test  data 
for  many  of  the  31  systems  identified  in  this  report.  However,  since  that  time,  added  clarity  in 
the  CENC  mission  critical  listings  and  synchronization  of  the  OSD  data  base  and  actual  Y2K 
Operational  Evaluations  (OPEVALS)  have  resulted  in  resolution  of  the  current  testing  status  of 
the  identified  31  communications  systems.  The  following  summary  provides  a  brief  overview  of 
the  resolved  status  of  the  31  systems.  Detailed  resolution  for  each  system  is  included  as  an 
attached  chart. 

Testing  completed:  14  of  the  31  systems  have  completed  their  required  Y2K  testing. 

Testing  planned:  One  (1)  system  (Senior  SPUR)  of  the  31  systems  now  has  a  test  planned  for 
October  15, 1999.  This  will  complete  the  required  testing  for  this  system. 

No  testing  required/Trusted:  Four  (4)  of  the  31  systems  do  not  require  testing  because  they 
have  been  identified  as  trusted  systems.  Trusted  systems  are  those  systems  that  cannot  have  their 
clocks  rolled  forward  into  a  Y2K  environment  for  evaluation  due  to  safety,  security,  or 
operational  necessity.  Most  of  these  systems  are  Satellite  systems  that  are  supporting  deployed 
forces. 

No  testing  required/Developmental  System:  Two  (2)  of  the  31  systems  do  not  require  testing 
because  they  are  developmental  and  will  not  be  deployed  before  the  millennium  rollover. 
However,  these  systems  are  identified  in  the  Y2K  Database.  They  are  currently  going  through 
the  five  phase  validation  process. 

No  testing  required/Does  process  dates:  Nine  (9)  of  the  31  systems  do  not  process  dates  and 
are  exempted  from  end-to-end  test  requirements. 

No  testing  requlred/Not  mission  critical:  One  (1)  system  has  been  re-classified  as  a  non¬ 
mission  critical  system  and  does  not  require  additional  tests. 
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GAO  DRAFT  REPORT  DATED  17  SEPTEMBER.1999  (GAO  CODE  511659)080  CASE  1696 
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Appendix  III 

Comments  From  the  Department  of  Defense 


OAO  DRAFT  REPORT  DATED  1 7  SEPTEMBER, 19M  {GAO  CODE  51165fl)OSD  CASE  1896 

DEFENSE  COMPUTERS:  'DOD  Y2K  FUNCTIONAL  END-TO-END  TESTING  PROGRESS  AND  TEST  EVENT  MANAGEMENT 


Non-Misalon  CrWca);  No 


I  I  'I  M 


Interoperable  Ajrfaome  [IADL _ |aS0064D6|  V  [testing  ragulrgd - 

NOTE:  1)  Dav(l)*DavelcpmanlaM  -  Brand  new  system  with  nowtunrttonaHty 

2)  Trusted  System:  Trusted  systems  are  those  that  cannot  have  docks  rolled  Into  a  Y2K  environment  for 
evaluation  due  to  safety,  security,  or  operational  reasons. 
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Ordering  Information 


The  first  copy  of  each  GAO  report  and  testimony  is  free. 

Additional  copies  are  $2  each.  Orders  should  be  sent  to  the 
following  address,  accompanied  by  a  check  or  money  order  made 
out  to  the  Superintendent  of  Documents,  when  necessary,  VISA  and 
MasterCard  credit  cards  are  accepted,  also. 

Orders  for  100  or  more  copies  to  be  mailed  to  a  single  address  are 
discounted  25  percent. 

Orders  by  mail: 

U.S.  General  Accounting  Office 
P.O.  Box  37050 
Washington,  DC  20013 

or  visit: 

Room  1100 

700  4th  St.  NW  (corner  of  4th  and  G  Sts.  NW) 

U.S.  General  Accounting  Office 
Washington,  DC 

Orders  may  also  be  placed  by  calling  (202)  512-6000 

or  by  using  fax  number  (202)  512-6061,  or  TDD  (202)  512-2537. 

Each  day,  GAO  issues  a  list  of  newly  available  reports  and 
testimony.  To  receive  facsimile  copies  of  the  daily  list  or  any  list 
from  the  past  30  days,  please  call  (202)  512-6000  using  a  touchtone 
phone.  A  recorded  menu  will  provide  information  on  now  to  obtain 
these  lists. 

For  information  on  how  to  access  GAO  reports  on  the  INTERNET, 
send  an  e-mail  message  with  “info”  in  the  body  to: 

info@www.gao.gov 

or  visit  GAO’s  World  Wide  Web  Home  Page  at: 
http://www.gao.gov 
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